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COMMUNICATION SYSTEM AND METHOD INCLUDING RICH MEDIA TOOLS 

BACKGROUND OF THE INVENTION 

The present invention relates to multimedia communications, and more 
particularly, to a rich media communication system based on user combinations of 
computer graphics, video, still images, audio, and text. 

Communication formats have evolved from text to interactive audio-visual 
communications. Recent virtual reality models generally involve the creation of a shared- 
space world in which the persons involved in the experience each represent a character in 
the shared-space world. The user representations, often referred to as avatars, in general 
consist of video representations or of graphic representations that are animated by a 
mouse or keyboard. Such virtual reality models often have been limited to real-time 
interactive communications. Such scenarios and representations limit the creativity and 
expression available using rich media communication formats. 

Accordingly, there exists a need for a system and related tools for providing rich 
media communications. The present invention satisfies these needs. 

SUMMARY OF THE INVENTION 

The present invention is directed to a system, and related method, having tools for 
generating and presenting rich media communications. The tools allow improved and 
lower cost rich media communications. 

The invention may be embodied in a rich media communication system having a 
theater and a player. The theater includes a representation associated with a person. The 
representation provides a choice of visual presence for the person. The player presents 
the theater at a remote location. 

An alternative embodiment is a rich media theater controller having a theater 
window and a presentation control. The theater window has a background presentation 
scene. The presentation control selects a character for a presentation in the theater. The 
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character presentation may be selected from an avatar, a blue screen cutout of the 
character, or a plain video presentation. 

The theater window may have a media target onto which the person may direct 
media, such as a video stream or audio, for display or play. The visual representation 
may be an avatar representation that is driven by visual sensing of the person. A rich 
media messaging directory service or an online directory may be used for locating other 
users capable of communicating with rich media presentations. The rich media 
communication system may include a status window indicating rich media 
communications received, the user's visibility to other users, the user's availability to 
other users, and the user's automatic response to rich media communication messages 
from other users. The rich media communication users may enter into user defined 
communities organized in hierarchical levels. Also, a theater window may be published 
to a rich media website. 

The rich media communication system may include a message center having a 
message reader. The message reader may have a text message window and a rich media 
presentation window. The rich media window may be toggled off such that a user may 
first read only a text message in the text message window before requesting transmission 
of a rich media message for presentation in the presentation window. 

In another embodiment, the rich media communication system includes a server 
infrastructure, a content client, and a plurality of communication clients. The server 
infrastructure provides web hosting, message hosting and communication services. The 
content client has an authoring tool for generating a rich media communication. The 
plurality of communicator clients display the rich media communication at remote 
locations using the server infrastructure. 

Other features and advantages of the present invention should be apparent from 
the following description of the preferred embodiments taken in conjunction with the 
accompanying drawings, which illustrate, by way of example, the principles of the 
invention. 
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BRIEF DESCRIPTION OF THE DRAWINGS 

Figure 1 is an image of an application window for a communication system 
having rich media tools in accordance with the present invention. 

Figure 2 is a image of a presentation control window for display in the application 
window of Figure 1, having a variety of rich media targets and communication channels 
for generating a personalized rich media communication. 

Figure 3 is an image of a player window for presenting a personalized rich media 
communication. 

Figure 4 is an image of a file manager window of theaters for use in the rich 
media communication system of Figure 1. 

Figure 5 is an image of a file manager window of avatars for use in the rich media 
communication system of Figure 1. 

Figure 6 is an image of a file manager window of rich media files for use in the 
rich media communication system of Figure 1 . 

Figure 7 is an image of a message center window of an inbox for use in the rich 
media communication system of Figure 1. 

Figure 8 is an image of a message center window of an outbox for use in the rich 
media communication system of Figure 1. 

Figure 9 is an image of a message reader window for presenting a personalized 
rich media communication provided as a message. 

Figure 10 is an image of an editor window for generating avatar images for use in 
the rich media communication system of the invention. 

Figure 11 is an image of a status window for use the in the rich media 
communication system of Figure 1 . 

Figure 12 is an image of a card editor window for use in the rich media 
communication system of Figure 1. 

Figure 1 3 is an image of an address book window for use in the rich media 
communication system of Figure 1 . 
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Figure 14 is an image of a directory window for use in the rich media 



communication system of Figure 1 . 

Figure 15 is an image of a directory window for use in the rich media 
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communication system of Figure 1. 

Figure 16 is a block diagram of the architecture of the rich media communication 



system of Figure 1 . 

Figure 17 is a block diagram of the architecture of a personal communicator of the 
system of Figure 16. 

Figures 18A and 18B show a block flow diagram of rich media communication 
1 0 across a communication channel in accordance with the invention. 



DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS 

The rich media communication system of the present invention provides a user 
with a three-dimensional (3D) communication space or theater having rich media 
15 functions. The user may be represented in the theater as a segmented video image or as 
an avatar. The user is also able to communicate by presenting images, videos, audio files, 
or text within the theater. The system may include tools for allowing lowered cost of 
animation, improved collaboration between users, presentation of episodic content, web 
casts, newscasts, infotainment, advertising, music clips, video conferencing, customer 



20 support, distance learning, advertising, social spaces, and interactive game shows and 
content. 

The communication system may use existing Internet protocols, communication 
channels and bandwidth reducing techniques to allow rich media communications over 
limited bandwidth channels. For example, a theater may be created which has an 
25 individualized space with media targets and an image or representation of the user. The 
theater space is transmitted to a receiver only once. The user representation may be 
represented by an avatar animated using low-bandwidth control signals, or alternatively, 
by a bandwidth reducing video cutout. 
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The rich media communication system of the invention allows each person to 
create his own world in which he is presented to others. The person's individual 
creativity may provide enhanced entertainment and attention retaining value to the 
communications space. 

The rich media communication system of the present invention may be embodied 
in an application window for a communicator shown in Figure 1 . The communicator 
window 10 is composed of a menu bar 12 and a non- interactive background 14. A 
variety of communication tools may be invoked in the application window, which tools 
arise as child windows on the background. The communicator's application window has 
three global menu options: file 16 , edit 18, and window 20. In addition to the global 
menu, a global toolbar may be used for the application window options. 

Under the file menu 16, a previously saved session file or environment may be 
opened 22, the current session may be saved 24, or the program quit 26. Under the edit 
menu 18 are context sensitive cut, copy, and paste. Also, an option exists for editing user 
preferences. Under the window menu 20 are options to open the variety of child window 
tools. As discussed in more detail below, the tools may include a player 28, a controller 
30, a chat module 32, a monitor 34, a status window 36, an editor 38, a message center 
40, directories 42, a card editor 44, an address book 46, preferences 48, a block list 50, a 
message reader 52, and a message composer 54. Through the window menu option, a 
communication tool may be invoked and an associated window pops up on the 
background. As shown in Figure 1, the player 28, the controller 30, the monitor 34, the 
message center 40, the address book 46, and the chat module 32 have been invoked. The 
individual tool windows may be positioned as desired by the user and may be placed in an 
overlapping format as is common to typical window applications. 

The controller window 30 is shown in Figure 2. The controller allows a user to 
control the content that is presented to other users as sessions or messages. The controller 
window may also show the user what other users may currently view. The controller 
window may be formatted with three buttons, 56, 58 and 60, that open file manager 
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windows. The three file manager windows are the theater file, the avatar file, and the 
media file manager windows. 

In the center of the controller 30 is a display region 62 that provides a working 
area for constructing a rich media communication session. The controller may provide a 
visual representation of a user using four buttons, 64, 66, 68 and 70, that gives the user 
media control of the user's appearance. The buttons are radio buttons that immediately 
switch between available visual modes. The modes include an avatar mode, a video 
mode, a cutout mode, or a show nothing mode. If a mode is not available in particular 
theater, the associated button is grayed out. 

The controller window 30 may be organized as a theater that provides a stage on 
which a representation 72 of the user may be displayed and provides media targets, 74 
and 76, on which selected media from a file manager (Figures 4-6) may be placed by the 
user. The theater may be selected or changed by selecting a theater from the theater file 
menu and dropping the theater into the controller presentation window. An avatar may be 
selected from the avatar file manager and dropped it onto the stage of the theater. In the 
theater, shown in Figure 2, the avatar mode has been selected. A user may select a media 
file from the media file manager and drag and place it onto a media target. The media 
may be a video, a still image, or audio file. When media is placed on a media target, it is 
either displayed or played in the media target. Media may be presented in the targets by 
selecting media from the media file manager and dropping it onto a particular media 
target or by dropping the media onto the theater. When a media file has dropped on the 
theater, the theater places the media into the first available mediate target as designated 
by the theater designer. If an audio file is dropped onto the stage, the audio is directed to 
a speaker icon 78 and the audio file played. If the audio file is dropped onto non-audio 
target, the audio file is nevertheless played. The media targets may be either 2-D or 3-D 
panes as specified by a particulars stage of the theater. A special media target is the 
speaker icon next to the presentation the window. This media target receives and plays 
audio information only. The user may right click the mouse over a media target to invoke 
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a pop-up menu having the following options: open file for opening a media file for 
display in that particular target, or properties for display of the media. For example, the 
display properties may include a stretch, crop, playback rate, or show all frames. 

The controller 30 also includes control buttons for controlling the media in the 
targets. The particular media target is selected by clicking on the target and its properties 
may be controlled using the control buttons. If no immediate target is currently selected, 
the media controls are grayed out. 

The controller 30 includes a camera button 86 which, when toggled, brings up a 
camera control user interface overlaid on top of the window. The controller may also 
include a loudspeaker button 88 that controls the volume of the audio coming from the 
user's microphone. The controller also includes a behavior button 90 which, when 
invoked, pops up a pop-up menu of all the available behaviors for a currently displayed 
avatar. The user may then select one of the behaviors for the avatar. The pop-up menu 
also shows any hot keys associated with particular behaviors. The controller also 
includes mutually exclusive radio buttons, 92 and 94, for move/select. When the select 
button is selected, a mouse click on an object causes the media controls to be directed to 
that target. When the move button is selected, a mouse click on an object brings up a 
move interface on top of the object. Also, object translation and rotation may be 
performed using mouse clicks and or drags. 

The controller 30 includes a recorder for savings of a user's session. The recorder 
includes buttons, 96, 98 and 100, for saving the recorded session as a file, making a 
message out of the recorded session, or making a web page out of the recorded session. 
The controller also includes a directory button 102, an address book button 104, and a 
friend's button 106. 

The monitor window 34 (Fig. 1) is generally used in conjunction with the 
controller. The monitor window is used to monitor the quality of data when the video cut 
out modes are selected, and the performance of the visual sensing software when the 
avatar mode is selected. The monitor window includes three radio buttons for choosing 
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between display of the live video coming from the user's camera, the live video associated 
with a blue screen cut out of the user, or the live video with sensing control points 
overlaid on it showing the performance of a tracker for animating the avatar. The 
monitor window also includes a volume slider for controlling the user's volume. 

The player window 28 is shown in Figure 3. Rich media presentations are 
delivered to a user's player for display. The player includes buttons, 108, 1 10, 1 12, 1 14 
and 1 16, for getting or showing a user's card, for privacy or blocking, for camera control, 
and a slider 1 18 for volume control. 

The file manager includes three buttons, 120, 122 and 124, for toggling the file 
manager between the theater file manager 126, the avatar file manager 128, and the media 
file manager 130. The theater file manager 126, shown in Figure 4, displays a thumbnail 
associated with each theater. A theater may have more than one configuration, and these 
configurations are collected into one object. The avatar file manager 128 is shown in 
Figure 5 and the media file manager 130 is shown in Figure 6. In the media file manager 
130, the user is shown an icon for every file in the current file directory which may be 
dropped onto media target. The icons or symbols indicate whether the file is a video, still 
image, or audio file. Media files may be dragged and dropped onto media targets in the 
controller window. The file manager also includes cut, copy, paste and delete option 
under an edit menu available in the media file manager. 

The message center window 40 is for handling rich media communication 
messages. The messages may be either just text, text with attached rich media, or only 
rich media. The inbox 132 of the message center is shown in Figure 7. The inbox and 
outbox may be selected using two radio buttons, 134 and 136, near the top of the message 
center. When the inbox button is selected, the message center window will show what is 
in the user's in box, and when the out box radio button is selected, the message window 
shows what is in the user's outbox. A new message button 138 opens a message 
composer window allowing the user to create a new message. A reply button 140 opens 
the message composer window with the sender of the currently selected message in the 
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"to" field. A connect button 142 allows the user to attempt to open a live connection with 
the sender of a currently selected message. If the live connection is successful, the sender 
will appear in a new player window 28. A delete button 144 permanently deletes the 
currently selected message. A block button 146 adds the sender of a currently selected 
message to a block list. The currently selected message is immediately deleted and future 
messages from the sender are also deleted. After a sender placed on the block list, the 
sender will not exist again in the inbox of the user. 

The message list 148 is a columnar table of all the messages that the user has 
received. Using a mouse to click on any of the column headings sorts the message list by 
that heading. Double clicking the mouse on the subject of the message opens that 
message in a new message reader window. If a rich media attachment is included in the 
message, an icon will be displayed in a media icon column. Rich media sessions which 
have been included as an attachment and which have been downloaded will appear as an 
icon in a file column. Rich media sessions which have not been downloaded may be 
marked for downloading in a download column. A triangle icon 150 associated with the 
words "download manager" is provided in the message center window 40. When the 
download manager triangle icon is selected, associated buttons are provided for allowing 
the user to control downloading of rich media to a local hard disk. A "get attachment" 
button 1 52 allows the user to download rich media associated with the currently selected 
message. A "get marked attachment" 154 button allows a user to download rich media in 
marked messages. A "get all attachments" button 156 allows the user to download any 
rich media which has not been downloaded. A stop button 158 allows user to halt the 
download process. A downloading message indicator 160 may show how much more of 
a rich media component is left for downloading both as a visual indicator and as a 
percentage. The current download speed also may be indicated. 

By selecting an outbox button 136 of the message center, as shown in Figure 8, 
the user is provided with a list of the messages that the user has sent to other users. A 
send another button 162 allows the user to send a message to additional users. This 
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button brings up a message composer with everything filled in with the selected message 
except the "to" field. An address button 164 allows the user to open an address book of 
received user cards. Buttons 166, 168, 170 and 172, are provided to play, file, delete and 
view messages. The other buttons of the message center performed as described above 
5 with respect to the inbox. 

^Qie message reader window 50 is a two-part window that allows a user to play 
messages. \\?!Ten4he reader is selected or a message is selected for playing, the first top 
portion 174 of the reade^dicates whom the message is from, the subject of the message, 
and associated message text^Tli^ize of the rich media component for downloading is 

10 also shown in the message window. The^ufcer may toggle the second half 176 of the 
reader window, which allows the user to display thel'feh.media portion of the message. 
When the rich media portion 1 76 is selected, the media player^ens in the bottom 
portion of the window and the rich media starts streaming and playing lJKhe media player 
window. A download indicator 178 shows the progress of the rich media downlo&el^ 

15 An editor window 38, shown in Figure 10, allows a user to create custom avatars, 

new theater stages, and to edit other media. The editor window includes five sub or child 
windows. Each child window is selected using an appropriate radio button 1 80 at the 
bottom of the editor window. Each child window has its own user interface that allows 
the user to add the associated media, avatar, or theater. The theater editor allows the user 

20 to create a custom theater. The user may change the texture mapped on parts of the 
theater, may bring in new objects. Alternatively, the media editor allows the user to 
launch helper applications that provide editing of video, audio, video with audio, and still 
pictures. 

v ^^— An^^vatar editor window allows a user to generate an avatar that looks and 
25 appears similar to theTi^^A^vizard allows the user to select a head geometry that is 

similar to the head geometry of the userT"A-new avatar is generated based on an image of 
the user. Alternatively an existing avatar may^beedite3""te Jjjesatisfaction or the user. 
The avatar may be animated or driven by wavelet-based facial mofion^apture as 
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described ini^Sr^atgi^pplication No.: 09/188,079 (WO 99/53443 Al), which is 
incorporated herein byref^encg^ 

The avatar window includes a wizard that leads the user through a sequence of 
steps for allowing the user to improve the accuracy of tracking of the avatar tracker. The 
5 avatar wizard may include a tutor face that prompts the user to make a number of 
expressions and varying head poses. A position is taken for each image and facial 
features are automatically located for each face image. The user has the opportunity to 
manually correct the positions of the automatically located features. 

The status window 36, shown in Figure 1 1, displays concise information 
10 regarding the user and the user's friends who are currently online. The status window 
shows whether the user has received new rich media messages and allows the user to 
select his online visibility. For example, the user may select to be visible on white and 
yellow pages, to current cardholders, friends, or the user may select invisibility. In 

HF 

addition the user may select to whom he is available. The availability may range from 
1 5 available to everyone, available for cardholders, available for friends, or unavailable. The 
user may also select how his automated call response will react to incoming calls. For 
example, the call response may be auto accept, caller ID, busy, or a message that the user 
will call right back. The field also shows a user's identity or ID. The concise nature of 
the status window information allows the status window to reside permanently on a users 
20 desktop. 

A user card editor window 38 is shown in Figure 12. User cards are a convenient 
means for allowing the users to exchange their coordinates and parameters. A user may 
appear under several different user IDs. For each ID, the user may provide the card to 
friends and other users. The user's ID generally includes the user's nickname, address, 
25 phone, email, a web page URL, and any comments that the user may provide. The card 
editor may also show thumbnail of the user in a rich media environment. A white pages 
button 182 allows the user to be listed in the white pages. A friends button 184 allows 
predetermined friends of the user to view whether the user is online. 
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An address book or card manager 46 is shown in Figure 13. The address book 
includes tabbed fields, 184 and 186, indicating to whom the user has distributed cards and 
from whom the user has received cards. Several columns indicate whether the cards are 
associated with friends, along with the card's user ID, nickname, and the date the card 
was received or sent. The address book has buttons for searching for nicknames or users 
188, connecting with a particular user 190, sending messages to users 192, sending cards 
to users 194, adding users to the address book 196, deleting users from an address book 
198, blocking users 200, and editing the user's properties 202. 

^^directory 42 associated with the rich media communication system is shown in 
10 Figures H^a^lS^The directory includes an online directory 204 associated with a 

hierarchal organization, a white page directory 206, and a yellow page directory 208. The 
online directory 204 allowsthe user to roam between the hierarchal levels. The user may 
be in only one location at a tim^the^^t they location are listed by their user ID or 
nickname. User may leave a card in order Yor^other users to leave a message for him after 
15 the user has left an area. A button in the director>^kxws the user to toggle between all 
members a location and live members at a location. A user "hiay directly contact a live 
member in the directory. ^ 

The hierarchal levels of the directory may be cities, neighborhoods, houses, and 
rooms. Cities and neighborhoods may be put in place by server staff. Houses and rooms 
20 may be built by users. Each hierarchal level has an information button. When activated, 
a rich media message plays describing the location. To build a house, a user clicks on a 
house builder button next to a neighborhood, and to build a room, a user clicks on a 
builder button next to a house. Rooms in houses may have bulletin boards upon which 
users may leave or read messages. Entering a room is similar to entering a conventional 
25 chat room. Each live member of the room or house appears as a thumbnail with his 
theater next to his name. 

A block diagram of an embodiment of a rich media communication system 210 of 
the invention is shown in Figure 16. The system includes a rich media server 212 and a 
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number of rich media clients 214. A user creating a rich media communication has a 
client that includes to a web browser 216, a personal communicator 218, and an authoring 
tool 220. The authoring tool includes a creator studio 222 that receives live input 224 and 
stored media 226. The live input may be from a camera, microphone, keyboard or 
mouse. The stored media may be in the form of video, audio, still images, or 3-D models. 
In addition, additional media may be obtained from a connection to the Internet 228 or 
from other consumer electronic devices 230. 

The server includes a web server 232, message hosting 234, and communication 
services 236. The server also includes resource information and parameters for each user 
allowing other users to engage in direct communication with a user based on the 
parameters stored on the server. 

Clients of other users may include a personal communicator 2 1 8 having only a 
player window or may include the authoring tools described above. A client side 
personal communicator 218 is shown in more detail in Figure 17. The communicator has 
links to the server 212 through message terminus 238, real time terminus 240, and a 
connector 242. Media is input from the live input 224 or from the stored media 230 into 
the authoring tool or personal creator 222. The media is encoded into a stream as a real 
time format or as a message format. In the real time format, the stream is directed to the 
communicators of other users. In the message format, the stream is directed to a message 
center 244 that stores the message 246 and provides a message composer 248 and 
message reader 250. 

A flow diagram of the rich media communications data or bit stream is shown in 
Figures 18A and 18B. Media is input into a multiplexer 252 which adds timestamp data 
and composes as a bit stream 254 after separating the data according to the data type 
features. The data includes information on the user's representation (video cutout or 
avatar animation values) and the user's audio and chat text messages. The multiplexer 
256 composes the bit stream and provides the bit stream to either a network transmit layer 
258 or an off-line message file 246. The transmitted bit stream is received by other users 
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and a demultiplexer 260 decomposes the received bit stream into separate data streams. 
The data streams are separated into individual media elements which are provided to 
render scheduler 262. The render scheduler provides the media elements to a player 28 or 
external renderer 264. 

Although the foregoing discloses the preferred embodiments of the present 
invention, it is understood that those skilled in the art may make various changes to the 
preferred embodiments without departing form the scope of the invention. The invention 
is defined only by the following claims. 
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